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ABSTRACT 

Objective To implement routine in-house monitoring 
of risk-adjusted 30-day mortality following paediatric 
cardiac surgery. 

Design Collaborative monitoring software development 
and implementation in three specialist centres. 
Patients and methods Analyses incorporated 2 years 
of data routinely audited by the National Institute of 
Cardiac Outcomes Research (NICOR). Exclusion criteria 
were patients over 16 or undergoing non-cardiac or only 
catheter procedures. We applied the partial risk adjustment 
in surgery (PRAiS) risk model for death within 30 days 
following surgery and generated variable life-adjusted 
display (VLAD) charts for each centre. These were shared 
with each clinical team and feedback was sought. 
Results Participating centres were Great Ormond Street 
Hospital, Evelina Children's Hospital and The Royal 
Hospital for Sick Children in Glasgow. Data captured all 
procedures performed between 1 January 2010 and 
31 December 201 1. This incorporated 2490 30-day 
episodes of care, 66 of which were associated with a 
death within 30 days.The VLAD charts generated for each 
centre displayed trends in outcomes benchmarked to 
recent national outcomes. All centres ended the 2-year 
period within four deaths from what would be expected. 
The VLAD charts were shared in multidisciplinary 
meetings and clinical teams reported that they were a 
useful addition to existing quality assurance initiatives. 
Each centre is continuing to use the prototype software to 
monitor their in-house surgical outcomes. 
Conclusions Timely and routine monitoring of risk- 
adjusted mortality following paediatric cardiac surgery is 
feasible. Close liaison with hospital data managers as 
well as clinicians was crucial to the success of the project. 



INTRODUCTION 

The recent Safe and Sustainable Paediatric Surgery 
Review of congenital heart services 1 recommended 
changes to the provision of paediatric cardiac 
surgery services in the UK. 2-4 The Review, 
managed by NHS Specialised Services, considered 
the delivery of congenital heart services by the 
NHS in England in response to concerns that 
smaller centres might not be in the best position to 
deliver sustainable 24-h care. In the absence of an 
accepted way to adjust for case mix, the review did 
not focus on outcomes but stressed in its recom- 
mendations the need for timely and meaningful 



reporting of mortality following paediatric cardiac 
surgery. 1 

Currently, UK paediatric cardiac surgery out- 
comes for individual procedure types are published 
online. 5 6 This represents an important contribu- 
tion to benchmarking of outcomes but limitations 
include an interval of at least a year before data are 
published, the small number of patients in many 
procedure categories and the omission of some pro- 
cedures from audit. Providing surgical centres with 
the means to monitor regularly their recent out- 
comes across an entire surgical programme can aid 
quality improvement. 7-13 For specialties with het- 
erogeneous case-mix, such monitoring should 
include risk adjustment. This is especially so for 
paediatric cardiac surgery where there have been 
recent advances in the number and types of avail- 
able treatment. Several reported methods for risk- 
stratification or adjustment 14-19 exist and Jacobs 
et al 20 recently used the STS-EACTS score 16 to 
compare the risk-adjusted outcomes of paediatric 
cardiac centres across North America. However, 
there have been no published reports of clinical 
teams using risk adjustment to monitor routinely 
their own programme-level outcomes in-house for 
the purpose of quality assurance. 

While academic research does often respond to 
clinical need, the rapid transfer from research 
to clinical practice is not common, delaying poten- 
tial benefit to patients and the clinical 
community. 7-10 13 In this paper, we describe the 
rapid implementation of a new risk model for 
paediatric cardiac surgery 19 in three UK centres to 
facilitate, for the first time in this specialty, regular 
monitoring of outcomes using routinely collected 
data. The risk model development finished in 
October 2011 and the study reported here ran 
from November 2011 to March 2012. 

METHODS 
Data 

Three UK paediatric cardiac centres were recruited 
to be part of this project via the Congenital Heart 
Disease Steering Committee of the Central Cardiac 
Audit Database. These centres were Great Ormond 
Street Hospital and Evelina Children's Hospital in 
London and The Royal Hospital for Sick Children 
in Glasgow. The centres volunteered to take part 
and cover the spectrum of unit sizes in the UK, from 
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the larger (Great Ormond Street) to the smaller (Glasgow). Each 
centre provided data on all paediatric cardiac surgery proce- 
dures 5 conducted in the period 1 January 2010 to 31 December 
2011 inclusive. In order to facilitate the process, data were sup- 
plied in the format that centres currently use for mandatory sub- 
missions for national audit. Records for patients over 16 years 
old, and patients who only underwent catheter procedures or 
non-cardiac procedures were excluded from analysis. 

Only pseudonymised, routinely collected data was used with 
no patient or carer interaction, so no ethical approval was 
required for this study. 

Analysis 

For each centre, every record of a procedure was allocated to a 
30-day episode of care. Each patient's first episode started with 
their first surgical procedure and the patient's vital status at 
30 days was assigned as a primary outcome. Any further proce- 
dures within this 30-day episode constituted a secondary 
outcome. A surgical procedure more than 30 days after the first 
procedure constituted the beginning of a new episode. 

An expected risk of death was estimated for each episode of 
care using the partial risk adjustment in surgery (PRAiS) 
model, 19 which partially adjusts for risk of death within 30 days 
following cardiac surgery in children under 16. The PRAiS 
model was built using a random 70% subset of 10 years of UK 
national audit data comprising 26 447 surgical episodes. It was 
then validated in the remaining 30% of the national dataset 
(10 597 episodes) which had not been used for model develop- 
ment. The risk factors included in the model are surgical pro- 
cedure, diagnosis, age, weight and co-morbidity. 

For each centre, risk-adjusted outcomes over time were dis- 
played using the variable life-adjusted display (VLAD) method 7 
that shows the difference between expected and observed mor- 
tality over time. In addition, further cardiac surgeries and inter- 
ventional catheterisations within each individual 30-day episode 
of care were displayed on the VLAD chart. 

To facilitate sustainable routine monitoring without specialist 
analytical support, we developed software implemented within 
Microsoft Excel to automate the preparation of data routinely 
supplied by centres for national audit for use with the risk 
model, and to generate and display VLAD charts. Since one aim 
was to provide centres with a 'ready to use' Excel spreadsheet 
for generation of VLAD charts, the software was developed in 
close collaboration with staff responsible for data submissions 
within the clinical centres, including repeated prototyping 
taking due account of existing data structures. 

Feedback from clinical teams 

The VLAD charts were presented to multidisciplinary clinical 
teams within each centre to elicit feedback on the displays, 
leading to subsequent iterative refinements by the research team. 
The first time VLAD charts were shown at each centre, the ana- 
lysts from University College London gave a brief introduction 
to the PRAiS risk model and VLAD chart methodology. This 
included highlighting known features of the risk model with 
respect to high risk cases and caveats around over-interpretation 
of VLAD charts without further investigation. The presentations 
of the VLAD charts on subsequent occasions were made by clin- 
icians to their normal audience of the regular mortality and 
morbidity meeting, since one of our aims was to ascertain 
whether this could be embedded into the usual routine quality 
assurance processes of each unit. The charts were shown at the 
participating sites on an approximately monthly basis. At these 
meetings comments were sought and received ad hoc 'from the 



floor'. These meetings are attended by a range of staff types; the 
number varies between 10 and 50 from centre to centre, and 
from month to month. The clinician responsible for presenting 
the VLAD chart and leading the discussion was debriefed by a 
member of the research team (KB, MU or CP) on at least three 
occasions. 

RESULTS 
VLAD charts 

Centres provided a total of 4192 records, 1543 of which were 
excluded because: patients were over 16 (244 records), patients 
had only catheter procedures (1242 records), the patients did 
not have a cardiac procedure (56 records) or it was a duplicate 
record (1 record). The remaining 2649 records corresponded to 
2490 30-day episodes, 66 of which were associated with a death 
within 30 days. 

VLAD charts for the three participating centres covering the 
2-year period of review are shown in figures 1-3. The risk 
model essentially benchmarks each centre's current short-term 
outcomes against UK outcomes achieved between 2007 and 
2010 (see Crowe et al 19 ) with partial adjustment for case mix. 
The plotted line goes up for each survival and down for each 
death. Over time, if outcomes are as expected based on the risk 
model, the end of the plotted line will tend to be close to zero. 
Given the low mortality associated with paediatric cardiac 
surgery (between 2007 and 2010, UK mortality was 2.9% 19 ), 
the plotted line will rise much less steeply for a run of survivors 
than it will fall for a run of deaths. The overall numbers of 
deaths and survivors are shown in the top left hand side of the 
VLAD charts. Re-interventions are displayed in the form of col- 
oured circles within the plots, such that their frequency can be 
reviewed by clinical teams alongside trends in risk-adjusted mor- 
tality. The three participating centres all ended the 2-year period 
within four survivors/deaths from what would be expected 
using the risk model but, importantly, the VLAD charts high- 
light different time periods for each centre that might have war- 
ranted further exploration. 

To help the reader interpret the VLAD charts, we have shown 
in figure 4 an enlarged section from the VLAD chart for centre 
B. Here the difference between lower- and higher-risk cases is 
evident: the plotted line for higher-risk patients who survived is 
steeper than for lower-risk survivors and vice versa for patients 
who died. 

We note that since calendar time is plotted on the horizontal 
axis, VLAD charts for centres with larger case volumes will have 
steeper slopes than VLAD charts for centres with a smaller case 
volume over the same period of time. Since the primary aim is 
for units to use this software for routine in-house monitoring 
and not for comparison with other units, this should not repre- 
sent a barrier to use. 

Clinician feedback and improvements to software 

Multidisciplinary teams in each centre reported that the VLAD 
charts showed useful information and that they prompted 
important discussion in mortality and morbidity meetings that 
in some instances led to further actions within the programme. 
Examples of this include: 

► The VLAD chart from centre A (figure 1) displays a long 
run of survivors during 2010 and a subsequent run with 
several deaths in the summer/autumn of 2011 leading to a 
downward trend in the VLAD chart. In respect of this 
cluster of deaths, certain patterns and learning points were 
found above and beyond those identified through the indi- 
vidual reviews of cases that had taken place. This specific 
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Figure 1 Variable life-adjusted 
display (VLAD) chart for centre A. Each 
dot represents a surgical episode. 



VLAD Chart from 04/01/2010 to 30/12/2011 



10 



Total number of 30-day survivors = 1156 
Total number of 30-day deaths = 25 
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learning in centre A included a focused multi-disciplinary 
audit of one particular procedure type (arterial shunt) and 
review of protocols for that patient group including case 
selection issues, technical aspects of the surgery and con- 
sideration of alternative procedures in certain patients. We 
note that national audit has not prompted concerns about 
this patient group (or any other) at centre A. 
► The section of the VLAD chart showing a downward 
trend in the summer of 2011 at centre B led to discussion 
within the surgical team at that centre about case-selection 
of patients for the Norwood operation. 
These examples demonstrate that the response of the multi- 
disciplinary team to adverse trends in the VLAD chart incorpo- 
rated practitioners from more than one discipline, and led to 
ongoing quality improvement initiatives that extended outside 
the meeting where the chart was shown. 



In addition to mortality outcomes, the secondary outcome of 
'reintervention within 30 days' is also shown in the VLAD 
charts by use of coloured dots. This can provide useful add- 
itional information to clinical teams reviewing these charts: for 
instance, although centre A has a long run of survivors in the 
summer of 2010 (figure 1), there were a cluster of reinterven- 
tions during that period. 

A review of the run of deaths shown by the VLAD chart from 
centre B (figure 2) during the first 6 months of 2011, while gen- 
erating useful discussion of the relevant cases, also raised the 
issue of inter-centre differences in case-mix and the known per- 
formance of the risk model. Clinicians considered that it would 
be informative to compare an individual centre's case mix to the 
national average and we modified the software to provide this 
option. An example is provided in figure 5, showing the break- 
down of case mix for centre B, alongside the comparator, which 



Figure 2 Variable life-adjusted 
display (VLAD) chart for centre B. Each 
dot represents a surgical episode. 



VLAD Chart from 04/01/2010 to 30/12/2011 



10 



Total number of 30-day survivors = 716 
Total number of 30-day deaths = 29 
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Figure 3 Variable life-adjusted 
display (VLAD) chart for centre C. Each 
dot represents a surgical episode. 
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represents the national audit data from 2007-2010. This 
centre's case mix contains higher proportions of medium and 
high risk cases than the national average. The comparison is par- 
ticularly relevant in this case, since the PRAiS risk model is 
known to underestimate risk slightly for the highest risk patients 
(with estimated risk above 10%), 19 so that the VLAD chart in 
figure 2 is likely to present a slightly pessimistic picture. 

Review of individual deaths highlighted the importance of 
co-morbid conditions and medical factors that were not fully 
accounted for in the predictions of expected risk. For example, 
prematurity was included in the risk model within a 'y es ' or ' no ' 
co-morbidity field indicating gestational age less than 37 weeks, 
as this is the definition used for national audit. 5 This does not 
capture the relative risk of operating at extremes of prematurity, 
for instance the case of a baby of only 25 weeks and weighing 



600 g. Also, certain serious co-morbidities, for example end 
organ damage prior to surgery, were not adequately described 
by the data submitted for national audit, despite appropriate 
fields being available. Finally, important but highly specific car- 
diovascular risk factors such as the aortic dimensions in candi- 
dates for the high risk Norwood operation 17 are not included in 
the routine dataset. 

The software package and data considerations 

Through working closely with the data managers, the software 
includes error checking and highlights obvious data anomalies 
for users. All required data processing is performed by the soft- 
ware to minimise the burden on centre data managers, so that 
only a straightforward 'copy and paste' from existing hospital 
databases into the Excel spreadsheet is required. 
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Figure 5 The Excel output showing the risk profile of centre B 
compared to the national average. CCAD, Central Cardie Audit 
Database. 



The prototype Excel software was given to each centre at the 
end of the implementation period and is now being used inde- 
pendently by those multidisciplinary teams to inform local gov- 
ernance processes and quality improvement initiatives. For 
instance, VLAD charts covering the previous 12 months are 
shown once a month in a multidisciplinary meeting at Great 
Ormond Street Hospital and any notable features discussed (for 
instance, a cluster of deaths or reoperations). 

DISCUSSION 

Our aim was to provide a means for units to monitor their own 
programme level outcomes for paediatric cardiac surgeries, 
benchmarking to recent national outcomes using a new model 
for risk adjustment and feeding these data back to the multidis- 
ciplinary team. We focused on team level outcomes to recognise 
the contribution made to care and decision making by all staff 
groups within the units. 

A new software package was developed that can be used by 
paediatric cardiac surgery centres to monitor their programme 
level, short-term outcomes with partial risk adjustment using 
the PRAiS risk model 19 and VLAD charts. 7 Presentation of 
VLAD charts to multidisciplinary meetings prompted useful dis- 
cussion of observed trends in outcomes. All three centres are 
using the software regularly to monitor their recent outcomes in 
the context of mortality and morbidity conferences. 

This routine in-house monitoring of risk-adjusted outcomes 
complements the annual national monitoring using funnel plots 
already in place for a range of individual surgical procedures on 
the National Institute for Cardiovascular Outcomes Research 
(NICOR) congenital heart disease web portal. The VLAD charts 
have obvious advantages in that first, the charts incorporate a 
unit's entire case load, thus increasing the chance of any concern- 
ing trend being detectable, and second, the VLAD charts can be 
delivered directly to clinical teams for very timely review of 
risk-adjusted outcomes with obvious advantages for quality 
assurance. 

The PRAiS risk adjustment model was developed in 2010- 
2011 using paediatric cardiac surgery audit data held by 
NICOR by a multidisciplinary group incorporating analysts and 
clinicians. The PRAiS risk adjustment model includes risk 
factors that have face validity and are already routinely collected 
for national audit: these attributes ensured that this method of 
risk adjustment was easy to implement in routine practice in the 



participating cardiac centres. Validation of PRAiS 19 showed that 
it performed as well as other published methods of risk adjust- 
ment 15 16 which increased its acceptability for use in monitoring 
among clinicians. There is the potential for future improved 
accuracy of risk ascertainment with better data quality, in par- 
ticular regarding co-morbidity information as discussed above. 

An ideal study exploring the introduction of a new monitor- 
ing system would involve only centres new to both the method 
of risk adjustment and the analytical team, to better mimic pro- 
spective national use. One of the centres (Great Ormond Street 
Hospital) co-developed the PRAiS risk model and has a long- 
standing productive collaboration with the analysts: this 
undoubtedly facilitated the success of the implementation in 
that centre. Close collaboration with data managers from the 
centres meant that the developed software could be responsive 
to centres' needs — for instance, the incorporation of reasonably 
comprehensive error checking and allowing for a variety of data 
input formats. The input of clinicians into the development of 
the software and pilot runs of the VLAD charts in a morbidity 
and mortality forum allowed the research team to add useful 
features to the output. We acknowledge that no formal system- 
atic evaluation of feedback was performed, rather that the clini- 
cians responsible for presenting the plots at their respective 
centres were informally debriefed by the research team. 

VLAD charts evaluating trends in the results of adult cardiac 
surgery have used data pertaining to individual operators 21-24 
and alternatively, programme level data. 7 The choice of pro- 
gramme based charts in this study reflects the fact that paediatric 
cardiac surgery demands high standards from cardiologists, 
nurses and intensive care specialists as well as surgeons as has 
been highlighted in recent service reviews. 1 25 26 Statistical 
methods based on CUSUM analysis have been used previously 
within VLAD charts to incorporate a 'signal' of poor outcomes. 27 
Rocket tails have been applied to VLAD charts to indicate gra- 
duations of likelihood that differences between expected and 
observed outcomes are due to chance. 28 Given the intended use 
of the paediatric cardiac surgery VLAD charts in a regular con- 
tinuous programme of review, we chose not to include statistical 
signalling in order to limit scope for complacency. We addition- 
ally discussed the potential for positive or negative trends to arise 
by chance with clinical teams and emphasised that VLAD charts 
represent a starting point for investigation, not an end. 

In addition to the points above, this study highlighted other 
factors that have implications for future implementation of 
routine monitoring in clinical practice. 

When initiating risk- adjusted monitoring of outcomes within 
centres, the discussions with clinical teams around context and 
intended use should consider the sensitivities involved. Previous 
barriers to clinician participation in reviews of patient safety and 
quality concerns included fear of censure and reprisals, lack of 
trust and concern regarding confidentiality. 29 Using the VLAD 
method for evaluation of recent outcomes means that on 
average, with a well calibrated risk model, a centre's outcomes 
will probably fluctuate around the zero line. We learned that clin- 
icians' first impressions of such plots can be that being close to 
the zero line is undesirable. This might be exacerbated by the fact 
that risk models currently used in adult cardiac surgery tend to 
overestimate risk of death so that VLAD charts of risk-adjusted 
outcomes in adult cardiac surgery normally show better perform- 
ance than expected from the risk model. 8 9 The VLAD charts do 
not evaluate the programme itself — quality of service is only one 
of many possible explanations for observed trends in a VLAD 
chart. Rather, the aim is to stimulate reflection on practice and 
assist a broader programme of continuous quality improvement. 
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An unanswered question is whether such routine monitoring 
will lead to service improvement in paediatric cardiac surgery. 
There is evidence to support this in adult heart surgery, 13 but 
with the management of congenital heart disease being far from 
a 'one-stop-shop', it seems likely that audit including review of 
short and longer-term outcomes will best serve the needs of this 
population. 

One final concern about the role of routine monitoring in 
surgical services arose during this study, which has relevance 
across many clinical areas. As hospital outcomes are increasingly 
published online and discussed in mainstream media, it is pos- 
sible that hospital centres would see a programme of routine 
monitoring of outcomes as a risk rather than an opportunity. 
On the other hand, publishing clinical outcomes can increase 
trust among service users and drive improvement across an 
organisation. 30 Perhaps further debate between clinicians and 
the public around the role of routine monitoring within service 
improvement is required. 
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